Summarizing Social Interactions Between Users

ABSTRACT

The disclosure includes a system and method for summarizing social interactions between users. The system includes a processor and a memory storing instructions that when executed cause the system to: receive a signal stream from at least one of a hardware sensor and a virtual detector, filter the signal stream and outputting filtered signal stream including data defining human-understandable actions, identify activities associated with a first user from the filtered signal stream, generate a summary of the first user&#39;s activities, determine that the first user is within proximity to a second user, determine a degree of separation between the first user and the second user in a social network, determine a time elapsed since a last interaction between the first user and the second user, classify the first user&#39;s relationship with the second user as being a first type of relationship, a second type of relationship or a third time of relationship, responsive to having the first type of relationship, generate a first summary for the first user that includes a notification that the second user is nearby, a last interaction with the second user and recent interactions with the second user, responsive to having the second type of relationship, generate a second summary for the first user that includes the notification that the second user is nearby, the last interaction with the second user and events that the first user and the second user share in common, and responsive to having the third type of relationship, generate a third summary for the first user that includes the notification that the second user is nearby and events that the first user and the second user share in common.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority, under 35 U.S.C. §119, of U.S.Provisional Patent Application No. 61/941,488, filed Feb. 19, 2014 andentitled “Summarizing Social Interactions Between Users,” which isincorporated by reference in its entirety.

BACKGROUND

The specification relates to managing user activities. Morespecifically, the specification relates to analyzing user activities andsummarizing social interactions between users.

Social settings can be very nerve wracking, especially when a personforgets how they know another person at the party. Current technologyexists to search for a person's name to see what has been posted on theInternet; however, this presumes that the person's name is known. Evenif the person's name is known, it can be hard to gather all thepertinent facts in the short time before speaking with the person.

SUMMARY

According to one innovative aspect of the subject matter described inthis disclosure, a system for summarizing social interactions betweenusers includes a processor and a memory storing instructions that, whenexecuted, cause the system to: receive a signal stream from at least oneof a hardware sensor and a virtual detector, filter the signal streamand outputting filtered signal stream including data defininghuman-understandable actions, identify activities associated with afirst user from the filtered signal stream, generate a summary of thefirst user's activities, determine that the first user is withinproximity to a second user, determine a degree of separation between thefirst user and the second user in a social network, determine a timeelapsed since a last interaction between the first user and the seconduser, classify the first user's relationship with the second user asbeing a first type of relationship, a second type of relationship or athird time of relationship, responsive to having the first type ofrelationship, generate a first summary for the first user that includesa notification that the second user is nearby, a last interaction withthe second user and recent interactions with the second user, responsiveto having the second type of relationship, generate a second summary forthe first user that includes the notification that the second user isnearby, the last interaction with the second user and events that thefirst user and the second user share in common, and responsive to havingthe third type of relationship, generate a third summary for the firstuser that includes the notification that the second user is nearby andevents that the first user and the second user share in common.

In general, another innovative aspect of the subject matter described inthis disclosure may be embodied in methods that include: receiving asignal stream from at least one of a hardware sensor and a virtualdetector, filtering the signal stream and outputting filtered signalstream including data defining human-understandable actions, identifyingactivities associated with a first user from the filtered signal stream,generating a summary of the first user's activities, determining thatthe first user is within proximity to a second user, determining adegree of separation between the first user and the second user in asocial network, determining a time elapsed since a last interactionbetween the first user and the second user, classifying the first user'srelationship with the second user as being a first type of relationship,a second type of relationship or a third time of relationship,responsive to having the first type of relationship, generating a firstsummary for the first user that includes a notification that the seconduser is nearby, a last interaction with the second user and recentinteractions with the second user, responsive to having the second typeof relationship, generating a second summary for the first user thatincludes the notification that the second user is nearby, the lastinteraction with the second user and events that the first user and thesecond user share in common, and responsive to having the third type ofrelationship, generating a third summary for the first user thatincludes the notification that the second user is nearby and events thatthe first user and the second user share in common.

Other aspects include corresponding methods, systems, apparatus, andcomputer program products for these and other innovative aspects.

These and other embodiments may each optionally include one or more ofthe following features. For instance, the operations include:determining closeness between the first user and the second user basedon at least one of the degree of separation and the time elapsed sincethe last interaction, and wherein classifying the first user'srelationship with the second user is based on the closeness; determiningwhat information of the summary to provide to the first user based onprivacy settings; and determining that the first user is withinproximity to the second user based at least in part on data receivedfrom at least one of the hardware sensor and the virtual detector. Forinstance, the features include: the first summary including action itemsthat the first user owes the second user; the second summary includingimportant events that occurred to the first user that the second usermight be interested in hearing about; the second summary including aname of a mutual connection and an event that the mutual connectionattended; the second summary including a recent post on a social networkthat was created by the second user; and the third type of relationshipbeing between a first user that has not met the second user in person.

The disclosure may be particularly advantageous in improving socialinteractions among people because a first user can get differentsummaries of user activities based upon different connections with otherusers that remind the first user about who the other users are, wherethey met last time, which topics they might discuss, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

The specification is illustrated by way of example, and not by way oflimitation in the figures of the accompanying drawings in which likereference numerals are used to refer to similar elements.

FIG. 1 is a block diagram illustrating an example of a system forgenerating a summary of a user.

FIG. 2 is a block diagram illustrating an example of a summaryapplication.

FIG. 3A is an example graphic representation of a user interface fordisplaying a summary where the first user and the second user are closefriends and/or interact with each other frequently.

FIGS. 3B and 3C are example graphic representation of user interfacesfor displaying a summary where the first user and the second user arefriends and/or interact with each other infrequently.

FIG. 3D is an example graphic representation of a user interface fordisplaying a summary where the first user and the second user arestrangers with information shared in common.

FIG. 4 is a flow diagram of an example of a method for generating asummary for a first user.

FIGS. 5A and 5B are flow diagrams of another example of a method forgenerating a summary for a first user depending on the type ofrelationship between the users.

DETAILED DESCRIPTION

The specification discloses a system and method for summarizing socialinteractions between users. The summary application receives a signalstream from at least one of a hardware sensor and a virtual detector.The summary application filters the signal stream and outputs a filteredsignal stream including data for defining one or morehuman-understandable actions. The summary application identifies one ormore activities associated with a first user from the filtered signalstream. The summary application generates a summary of the first user'sactivities. For example, the first user attended a conference, postedpictures of an important event, and checked-in at a restaurant. Thesummary application determines that the first user is within proximityto a second user and determines a degree of separation between the firstuser and the second user in a social network. For example, if the firstuser follows the second user they have a first degree relationship. Thesummary application determines a time elapsed since a last interactionbetween a first user and a second user. For example, if more than amonth has passed since they last interacted, they are not close friends.

The summary application classifies the first user's relationship withthe second user as being a first type of relationship, a second type ofrelationship or a third type of relationship. Responsive to having thefirst type of relationship, the summary application generates a firstsummary for the first user that includes a notification that the seconduser is nearby, a last interaction with the second user and recentinteractions with the second user. The first type of relationshipincludes, for example, friendship. Responsive to having the second typeof relationship, the summary application generates a second summary forthe first user that includes the notification that the second user isnearby, the last interaction with the second user and events that thefirst user and the second user share in common. This applies when theusers are acquaintances. For example, the first and second users havemutual friends and both lived in Cambridge at one time. In someembodiments, the summary application also generates a summary of all theimportant life events that occurred to the first user since he lastspoke with the second user. For example, the first user started a newjob and had a baby. Responsive to having the third type of relationship,the summary application generates a third summary for the first userthat includes the notification that the second user is nearby and eventsthat the first user and the second user share in common. For example,the first user and second user both attended the same conference lastweek. This is for users that do not know each other very well and givesthem things to discuss.

FIG. 1 illustrates a block diagram of a system 100 for summarizingsocial interactions between users. The illustrated description of thesystem 100 includes user devices 115 a . . . 115 n that are accessed byusers 125 a . . . 125 n, one or more social network servers 101 and anevent server 107. In the illustrated embodiment, these entities of thesystem 100 are communicatively coupled via a network 105. In FIG. 1 andthe remaining figures, a letter after a reference number, for example“115 a” is a reference to the element having that particular referencenumber. A reference number in the text without a following letter, forexample “115,” is a general reference to any or all instances of theelement bearing that reference number.

The network 105 can be a conventional type network, wired or wireless,and may have any number of configurations for example a starconfiguration, token ring configuration or other configurations known tothose skilled in the art. Furthermore, the network 105 may comprise alocal area network (LAN), a wide area network (WAN) (e.g., theInternet), and/or any other interconnected data path across whichmultiple devices may communicate. In some embodiments, the network 105may be a peer-to-peer network. The network 105 may also be coupled to orincludes portions of a telecommunications network for sending data in avariety of different communication protocols. In other embodiments, thenetwork 105 includes Bluetooth communication networks or a cellularcommunications network for sending and receiving data for example viaSMS/MMS, hypertext transfer protocol (HTTP), direct data connection,WAP, e-mail, etc. While only one network 105 is illustrated, in practiceone or more networks 105 may be coupled to the above mentioned entities.

The social network server 101 can be a hardware server that includes aprocessor, a memory and network communication capabilities. The socialnetwork server 101 is communicatively coupled to the network 105 viasignal line 102. In some embodiments, the social network server 101sends and receives data to and from one or more of the user devices 115a, 115 n and the event server 107 via the network 105. The socialnetwork server 101 includes a social network application 109 and adatabase 199.

A social network can be a type of social structure where the users maybe connected by a common feature. The common feature includesrelationships/connections, e.g., friendship, family, work, an interest,etc. The common features may be provided by one or more socialnetworking systems including explicitly defined relationships andrelationships implied by social connections with other online users,where the relationships form a social graph. In some examples, thesocial graph can reflect a mapping of these users and how they can berelated. The social network application 109 in the social network server101 manages the social network by handling registration of users,publication of content (e.g. posts, comments, photos, links, check-ins,etc.), hosting multi-user communication sessions, managing of groups,managing different sharing levels, updating the social graph, etc. Thesocial network application 109 registers a user by receiving informationsuch as a username and password and generates a user profile that isassociated with the user and stored as part of the social graph. In someembodiments, the user profile includes additional information about theuser including interests (e.g. soccer, reading, food, subscriptions,etc.), activities (e.g. search history, indications of approval, posts,comments, multi-user communication sessions, etc.), demographics (e.g.age, ethnicity, location, etc.) and profile rating and reputation (e.g.,intelligence rating, humor rating, etc.). The database 199 in the socialnetwork server 101 stores social network data associated with the users.For example, the database 199 stores social network data describing oneor more of user profiles, posts, comments, videos, audio files, images,sharings, acknowledgements, etc., published on a social network. Thesystem 100 may include multiple social network servers 101 that includetraditional social network servers, email servers, micro-blog servers,blog servers, forum servers, message servers, etc.

Furthermore, the social network server 101 and the social networkapplication 109 may be representative of one social network and thatthere may be multiple social networks coupled to the network 105, eachhaving its own server, application and social graph. For example, afirst social network may be more directed to business networking, asecond may be more directed to or centered on academics, a third may bemore directed to local business, a fourth may be directed to dating andothers may be of general interest or a specific focus.

The user devices 115 a, 115 n in FIG. 1 are used by way of example.Although only two user devices 115 are illustrated, the disclosureapplies to a system architecture having any number of user devices 115available to any number of users 125. In the illustrated implementation,the user 125 a interacts with the user device 115 a. In someembodiments, the summary application 103 a can be stored on the userdevice 115 a which is communicatively coupled to the network 105 viasignal line 108. The user 125 n interacts with the user device 115 n.The user device 115 n is communicatively coupled to the network 105 viasignal line 110.

In some embodiments, the user device 115 can be any computing devicethat includes a memory and a processor. For example, the user devices115 can be a laptop computer, a desktop computer, a tablet computer, amobile telephone, a personal digital assistant, a mobile email device, aportable game player, a portable music player, a television with one ormore processors embedded therein or coupled thereto or any otherelectronic device capable of accessing the network 105, etc.

In some embodiments, the user device 115 can include a mobile devicethat is worn by the user 125. For example, the user device 115 isincluded as part of a clip (e.g., a wristband), as part of a jewelry oras part of a pair of glasses. In another example, the user device 115can be a smart watch. The user 125 can view notifications from thesummary application 103 on a display of the device worn by the user 125.For example, the user 125 can view the notifications on a display of asmart watch or a smart wristband. In another example, the user 125 canview the notifications on an optical head-mounted display of a pair ofglasses. The user 125 may also configure what types of notifications tobe displayed on the device worn by the user 125. For example, the user125 may configure the wearable device to flash a LED light for fiveseconds if a friend's mobile device is detected in proximity to the user125.

In some embodiments, the summary application 103 can be split into somecomponents that are stored on the user device 115 a and some componentsthat are stored on the event server 107. For example, the summaryapplication 103 a on the user device 115 a acts in part as a thin-clientapplication and sends an event stream including one or more eventsassociated with a user to the summary application 103 b on the eventserver 107. The summary application 103 b on the event server 107augments the event stream by including new events and sends back theupdated event stream to the summary application 103 a on the user device115 a for presenting the event stream to the user 125 a.

In some embodiments, the summary application 103 b can be stored on anevent server 107, which is connected to the network 105 via signal line104. In some embodiments, the event server 107 can be a hardware serverthat includes a processor, a memory and network communicationcapabilities. The event server 107 sends and receives data to and fromother entities of the system 100 via the network 105. While FIG. 1illustrates one event server 107, the system 100 may include one or moreevent servers 107.

The summary application 103 can be software including routines forgenerating a summary of user activities. In some embodiments, thesummary application 103 can be implemented using hardware including afield-programmable gate array (FPGA) or an application-specificintegrated circuit (ASIC). In some other embodiments, the summaryapplication 103 can be implemented using a combination of hardware andsoftware. In some embodiments, the summary application 103 may be storedin a combination of the devices and servers, or in one of the devices orservers. The summary application 103 is described in further detailbelow with reference to FIG. 2.

The summary application 103 identifies activities associated with usersand generates a summary of the user activities for a user. In someembodiments, the summary application 103 determines that a first user iswithin proximity of a second user, generates a summary based on therelationship between the first and second users and provides the firstuser with the summary. The summary includes a notification that thesecond user is nearby, a last time the first user interacted with thesecond user and information about the at least one of the first user andthe second user. For example, if Joe came to Salt Lake City where Amylives after they last met with each other in their mutual friend Mary'shouse three months ago, the summary application 103 determines that Joeis an acquaintance of Amy (because they have not connected with eachother for the past three months) and generates a summary for Amy tonotify that Joe is nearby and that their last interaction was in Mary'shouse three months ago. The summary also includes a picture of Joelooking at the Seagull Monument in Salt Lake City, a picture of Amy'snew house and a picture of Amy having lunch with Mary. The summaryreminds Amy of Joe and provides topics that they can discuss (e.g., theSeagull Monument in Salt Lake City, Amy's new house or their mutualfriend Mary). As a result, the connection between Amy and Joe might beimproved.

The summary application 103 generates different summaries for a userbased on different relationships between the user and other users. Forexample, if the summary application 103 determines that Ryan is a closefriend of Richard since they talk on a social network every week. WhenRyan is attending a conference in the city where Richard lives, thesummary application 103 detects Ryan's location and generates a summaryto notify Richard that Ryan is nearby and remind Richard that he toldRyan that they will visit a neighboring national park together when Ryancomes to the city. In the meantime, if Oscar is also attending theconference in the city and he is not a close friend of Richard, e.g.,Oscar and Richard have no personal connections except that they went tothe same high school, the summary application 103 may generate adifferent summary that includes the information “both of you attendedMurray High School” for Richard responsive to Oscar attending theconference in the city.

Referring now to FIG. 2, an example of the summary application 103 isshown in more detail. FIG. 2 is a block diagram of a computing device200 that includes the summary application 103, a processor 235, a memory237, a communication unit 241, a storage device 243 and one or morehardware sensors 252 a . . . 252 n according to some examples. Thecomponents of the computing device 200 are communicatively coupled by abus 220. In some embodiments, the computing device 200 can be one of auser device 115 and an event server 107.

The processor 235 includes an arithmetic logic unit, a microprocessor, ageneral-purpose controller or some other processor array to performcomputations and provide electronic display signals to a display device.The processor 235 is coupled to the bus 220 via signal line 236 forcommunication with the other components. Processor 235 may process datasignals and may comprise various computing architectures including acomplex instruction set computer (CISC) architecture, a reducedinstruction set computer (RISC) architecture, or an architectureimplementing a combination of instruction sets. Although only a singleprocessor is shown in FIG. 2A, multiple processors 235 may be included.The processing capability may be limited to supporting the display ofimages and the capture and transmission of images. The processingcapability might be enough to perform more complex tasks, includingvarious types of feature extraction and sampling. In practice, otherprocessors, operating systems, sensors, displays and physicalconfigurations are possible.

The memory 237 stores instructions and/or data that may be executed byprocessor 235. The memory 237 is coupled to the bus 220 via signal line238 for communication with the other components. The instructions and/ordata may include code for performing any and/or all of the techniquesdescribed herein. The memory 237 may be a dynamic random access memory(DRAM) device, a static random access memory (SRAM) device, flash memoryor some other memory device known in the art. In some embodiments, thememory 237 also includes a non-volatile memory or similar permanentstorage device and media for example a hard disk drive, a CD-ROM device,a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memorydevice, or some other mass storage device known in the art for storinginformation on a more permanent basis.

The communication unit 241 transmits and receives data to and from atleast one of the user device 115, the event server 107 and the socialnetwork server 101 depending upon where the summary application 103 isstored. The communication unit 241 is coupled to the bus 220 via signalline 242. In some embodiments, the communication unit 241 includes aport for direct physical connection to the network 105 or to anothercommunication channel. For example, the communication unit 241 includesa USB, SD, CAT-5 or similar port for wired communication with the userdevice 115. In other embodiments, the communication unit 241 includes awireless transceiver for exchanging data with the user device 115 or anyother communication channel using one or more wireless communicationmethods, such as IEEE 802.11, IEEE 802.16, BLUETOOTH® or anothersuitable wireless communication method.

In some embodiments, the communication unit 241 includes a cellularcommunications transceiver for sending and receiving data over acellular communications network such as via short messaging service(SMS), multimedia messaging service (MMS), hypertext transfer protocol(HTTP), direct data connection, WAP, e-mail or another suitable type ofelectronic communication. In other embodiments, the communication unit241 includes a wired port and a wireless transceiver. The communicationunit 241 also provides other conventional connections to the network fordistribution of files and/or media objects using standard networkprotocols such as TCP/IP, HTTP, HTTPS and SMTP as will be understood tothose skilled in the art.

The storage device 243 can be a non-transitory memory that temporarilystores data used by the summary application 103, for example, a cache.The storage device 243 may be a dynamic random access memory (DRAM)device, a static random access memory (SRAM) device, flash memory orsome other memory device known in the art. In some embodiments, thestorage device 243 also includes a non-volatile memory or similarpermanent storage device and media such as a hard disk drive, a CD-ROMdevice, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flashmemory device, or some other mass storage device known in the art forstoring information on a more permanent basis. In the illustratedembodiment, the storage device 243 is communicatively coupled by the bus220 for communication with the other components of the computing device200 via signal line 240. Although only one storage device 243 is shownin FIG. 2A, multiple storage devices 243 may be included. In otherembodiments, the storage device 243 may not be included in the userdevice 115 and can be communicatively coupled to the user device 115 viathe network 105.

In the illustrated embodiment, the storage device 243 stores one or moreof raw data, signal streams, activities performed by one or more usersand analytics data associated with the activities. The data stored inthe storage device 243 is described below in more detail. In someembodiments, the storage device 243 may store other data for providingthe functionality described herein.

The hardware sensors 252 a . . . 252 n are physical sensors fordetecting data. Example hardware sensors 252 include, but are notlimited to, an infrared sensor, an accelerometer, a pedometer, a globalpositioning system (GPS) sensor, a Bluetooth sensor, a power detector, abattery detector, a camera, a light detection and ranging (LIDAR)sensor, a motion sensor, a capacitive sensor, a thermostat and amicrophone, etc. Other example hardware sensors 252 are possible. Thehardware sensor 252 a is communicatively coupled to the bus 220 viasignal line 251, and the hardware sensor 252 n is communicativelycoupled to the bus 220 via signal line 253.

In some embodiments, the one or more hardware sensors 252 generatesensor data and send the sensor data to a processing unit 204 of thesummary application 103. The sensor data generated by the one or morehardware sensors 252 are referred to as hardware raw data. Examplehardware raw data includes, but is not limited to, data describing anumber of steps from a pedometer, data describing a geographic location(e.g., a latitude, a longitude and an elevation of a location) and avelocity from a GPS sensor, data describing a presence of other devicesin close proximity to the user device 115 from a Bluetooth sensor, datadescribing a movement from an accelerometer (for e.g., the user device115 is being held in a certain orientation while watching a video,playing a video game, etc.), data describing brightness in anenvironment from a light detector, data describing detecting ambientsounds from a microphone, data describing detecting wireless accesspoints from wireless transceivers, etc. Other example hardware raw datais possible. In some embodiments, the one or more hardware sensors 252generate the hardware raw data with permission from the user and storethe hardware raw data in the storage device 243.

In the illustrated implementation shown in FIG. 2, the summaryapplication 103 includes a virtual detector 202, a processing unit 204,a filter engine 206, an activity identifier 208, an aggregator 210, asummarizing engine 212, a user interface engine 214 and a privacy engine216.

The virtual detector 202 can be software including routines forgenerating raw data. In some embodiments, the virtual detector 202 canbe a set of instructions executable by the processor 235 to provide thefunctionality described below for generating raw data. In someembodiments, the virtual detector 202 can be stored in the memory 237 ofthe computing device 200 and can be accessible and executable by theprocessor 235. The virtual detector 202 may be adapted for cooperationand communication with the processor 235 and other components of thecomputing device 200 via signal line 230.

In some embodiments, the one or more hardware sensors 252 generatehardware raw data, and send the hardware raw data to the processing unit204. The virtual detector 202 generates other raw data that is notrelated to hardware sensors 252, and sends the other raw data to theprocessing unit 204. The other raw data generated by the virtualdetector 202 is referred to as virtual raw data. In some embodiments,the virtual detector 202 generates the virtual raw data with permissionfrom the user.

Example virtual raw data includes, but is not limited to, software rawdata related to software stored on the user device 115, mobile networkinformation related to the user device's 115 mobile network, file statuson the user device 115, data describing interactions between the userand the user device 115 (e.g., the user turning up or turning downvolume, brightness, contrast, etc. on the user device 115, the userzooming in or zooming out of content displayed on the user device 115,the user scrolling down on a touch screen or typing in a user interface,the user making a phone call using the user device 115, etc.), datadescribing user interactions on a social network (e.g., the user viewinga social stream on a social network; the user publishing a post, sharinga web page, posting a comment, viewing a video, listening to an audiofile, playing an online game, submitting a survey, adding users as hisor her connections, etc., on the social network), the user's onlinesearch history, the user's browsing history and the user's communicationhistory (e.g., text messages, emails, etc.). In some embodiments, thevirtual raw data is retrieved with permission from the user, etc. Otherexample virtual raw data is possible. In some embodiments, the virtualraw data includes metadata associated with the user device 115.

Example software raw data related to software stored on the user device115 includes, but is not limited to, operating system informationrelated to the user device 115 (e.g., the user updating the operatingsystem, switching the operating system, etc.), applications stored onthe user device 115 (e.g., applications for fitness tracking, countingcalories, mobile payment, reading books, listening to music, etc.) andapplication usage information on the user device 115 (e.g., the userentering his or her gym routine into a fitness tracking application,opening a song playlist in a media library, closing an instant messagingapplication, deleting an unused application, updating an existingapplication, installing a new application, configuring an applicationsetting, etc.). Other example software raw data is possible. In someembodiments, the virtual detector 202 stores the virtual raw data in thestorage device 243.

The processing unit 204 can be software including routines for receivingsignal streams from the virtual detector 202 and/or one or more hardwaresensors 252. In some embodiments, the processing unit 204 can be a setof instructions executable by the processor 235 to provide thefunctionality described below for receiving signal streams from thevirtual detector 202 and/or one or more hardware sensors 252. In someembodiments, the processing unit 204 can be stored in the memory 237 ofthe computing device 200 and can be accessible and executable by theprocessor 235. The processing unit 204 may be adapted for cooperationand communication with the processor 235 and other components of thecomputing device 200 via signal line 232.

In some embodiments, the processing unit 204 receives a signal streamfrom the virtual detector 202, where the signal stream includes virtualraw data generated by the virtual detector 202. In other embodiments,the processing unit 204 receives a signal stream from one or morehardware sensors 252, where the signal stream includes hardware raw datagenerated by the one or more hardware sensors 252. In some otherembodiments, the processing unit 204 receives a stream of virtual rawdata from the virtual detector 202 and a stream of hardware raw datafrom the one or more hardware sensors 252, where the stream of virtualraw data and the stream of hardware raw data together form aconsolidated signal stream. The processing unit 204 sends the signalstream to the filter engine 206. In some embodiments, the processingunit 204 stores the signal stream in the storage 243.

In some embodiments, the processing unit 204 validates the data in thesignal stream for its usefulness. In some embodiments, the processingunit 204 saves a data block from the signal stream that indicates achange in state as when compared to a previous data block. For example,at a first timestamp, the processing unit 204 may receive a first set oflocation data from a GPS sensor indicating a user has just arrived at acoffee shop after coming out of a subway station, and the processingunit 204 may save the first set of location data. At a second timestamp,if the processing unit 204 receives, from the GPS sensor, a second setof location data which is identical to the first set of location data,indicating the user is at the same location as the first timestamp, theprocessing unit 204 does not save the second set of location data.However, at a third timestamp, if the processing unit 204 receives, fromthe GPS sensor, a third set of location data which is different from thesecond set of location data, indicating the user has left the coffeeshop and is now in the office, the processing unit 204 saves the thirdset of location data. At the first timestamp, the user is in transitsince the user just arrived at the coffee shop after coming out of asubway station; at the second timestamp, the user is stationary sincethe user is still at the coffee shop; at the third timestamp, the useris in transit again since the user has left the coffee shop and is nowin the office. The processing unit 204 saves data related to the transitmoments and ignores data related to the stationary moments.

In some examples, the processing unit 204 saves the data from the signalstream that indicate a change in a frequency of steps (for e.g., datafrom accelerometer), a change of velocity (for e.g., data from GPSsensor), a change of location (for e.g., data from a GPS sensor, awireless transceiver, etc.), a change in application usage (e.g., anapplication being opened, used, closed, updated, installed, etc.), achange in actions performed on a social network (e.g., a user loggingin, logging out, uploading a photograph, accepting invites, posting acomment, indicating an acknowledgement, adding other users asconnections, etc.), a change related to detecting a presence of otheruser devices 115 n in close proximity of the user device 115 a or otherchanges in state.

The filter engine 206 can be software including routines for filteringsignal streams. In some embodiments, the filter engine 206 can be a setof instructions executable by the processor 235 to provide thefunctionality described below for filtering signal streams. In someembodiments, the filter engine 206 can be stored in the memory 237 ofthe computing device 200 and can be accessible and executable by theprocessor 235. The filter engine 206 may be adapted for cooperation andcommunication with the processor 235 and other components of thecomputing device 200 via signal line 234.

In some embodiments, the filter engine 206 filters the signal stream todefine one or more human-understandable actions. For example, the filterengine 206 filters the signal stream to retrieve data describing anumber of steps from the accelerometer of the user device 115 andoutputs a filtered signal stream including step data. In anotherexample, the filter engine 206 filters the signal stream to retrievesequence of location and velocity data from a GPS sensor of the userdevice 115 and outputs a filtered signal stream including location data.In yet another example, the filter engine 206 filters the signal streamto retrieve data describing detection of a mobile device in closeproximity to the user device 115 and outputs a filtered signal streamincluding detection data. Such a filtered signal stream includes hashedidentifiers (i.e. hashed using phone number, email, or social networkprofile identifiers, etc.) associated with the mobile device in closeproximity of the user device 115.

In some embodiments, the filter engine 206 filters the signal stream tocombine different types of data in a filtered signal stream to defineone or more human understandable actions. For example, the filter engine206 outputs a filtered signal stream that combines one or more of thefollowing data including: (1) location and velocity data from a GPSsensor, and (2) detection data indicating presence of an automobile(e.g., Bluetooth enabled) and a mobile in close proximity, etc. toindicate travelling together with another user. In another example, thefilter engine 206 outputs a filtered signal stream that combines one ormore of the following data including: (1) ambient sound data from amicrophone, (2) location data from a GPS sensor or Wi-Fi access point,and (3) uploading one or more pictures with GPS tags matching thelocation data to the social network, etc. to indicate a socialgathering. In yet another example, the filter engine 206 outputs afiltered signal stream that combines one or more of the following dataincluding: (1) motion data from an accelerometer, (2) ambientillumination data from a light sensor, (3) energy usage data from apower detector on the user device 115, and (4) application usage datafrom an application manager in the user device 115, etc. to indicatesleeping or active day time activity.

In some embodiments, the filter engine 206 filters the signal stream toidentify changes in one or more human understandable actions. Forexample, assume a Bluetooth sensor on a user's mobile device isdetecting a presence of a number of mobile devices in close proximity ofthe user every five minutes from 1:00 PM to 1:30 PM. The filter engine206 filters the data generated by the Bluetooth sensor and outputs afiltered signal stream that includes (1) data indicating detection of afirst mobile device and a second mobile device in proximity of the userat 1:00 PM, and (2) data indicating detection that the second mobiledevice is no longer in proximity of the user at 1:25 PM. In anotherexample, assume a GPS sensor on a user's mobile device updates thelocation of the user every 2 minutes from 8:00 AM to 8:30 AM. The filterengine 206 filters the data generated by the GPS sensor and outputs afiltered signal stream that includes (1) a first set of location andtimestamp describing that the user arrived at a coffee shop at 8:04 AMand (2) a second set of location and timestamp data describing that theuser left the coffee shop at 8:30 AM. Other sets of location andtimestamp data received from the GPS sensor between 8:00 AM and 8:30 AMare not included in the filtered signal stream because they areidentical or too similar.

In some embodiments, the filtered signal stream includes data describingappearance and disappearance of another user device 115. For example,assume a Bluetooth sensor detects a presence of a friend's mobile deviceand generates data describing the presence of the friend's mobile deviceevery five minutes from 1:00 PM to 1:30 PM. The filter engine 206filters the data generated by the Bluetooth sensor, and outputs afiltered signal stream that only includes (1) data indicating anappearance of the friend's mobile device at 1:00 PM and (2) dataindicating the friend's mobile device was last detected at 1:30 PM. Insome other embodiments, the filtered signal stream includes dataindicating a change of a frequency of steps, a change of velocity, achange of application usage (e.g., an application being open or beingclosed), a change of actions on a social network (e.g., a user loggingin or exiting from a social network account) or other changes in actions

In some embodiments, the filter engine 206 filters the signal stream toinclude data from a Bluetooth sensor associated with a first user device115 of a first user, where the data can be used to determine a presenceof a second user device 115 that also has a Bluetooth sensor. Forexample, if the first user device 115 and the second device 115 are inproximity, the first user device 115's Bluetooth sensor generates dataindicating a presence of the second user device 115. In some examples,the data indicating presence of the second user device 115 can alsoindicate a presence of a second user associated with the second userdevice 115 (e.g., the first user and the second user are in proximity).For example, if the second user device 115 is a mobile device, thepresence of the mobile device may indicate the presence of the seconduser. In some embodiments, the filter engine 206 filters the signalstream to additionally include received signal strength indicator (RSSI)data from the Bluetooth sensor for increased granularity.

The detection using the Bluetooth sensors is easy to implement and canbe turned on automatically with the user's permission. If the two userdevices 115 are within the detection radius of the Bluetooth sensors,the detection yields accurate results. However, some devices may not beable to be discovered by Bluetooth sensors. For example, an old devicemay not be detected by a user device 115 having Bluetooth sensors.

In some embodiments, the filter engine 206 filters a first signalstream, and outputs a first filtered signal stream that includes a firstset of data from a first Bluetooth sensor associated with a first userdevice 115 of a first user. The first set of data indicates the firstuser device 115 detects a presence of a third device at a firsttimestamp. Also, the filter engine 206 filters a second signal stream,and outputs a second filtered signal stream that includes a second setof data from a second Bluetooth sensor associated with a second userdevice 115 of a second user. The second set of data indicates the seconduser device 115 detects a presence of the third device at a secondtimestamp. If the time difference between the first timestamp and thesecond timestamp is within a predetermined threshold (e.g., fiveseconds), the first set of data and the second set of data can be usedby the activity identifier 208 to determine that the first user device115 and the second user device 115 are in proximity since both of thetwo user devices 115 detect the third device within a short time period.The activity identifier 208 is described below in more detail.

For example, assume the third device is a vehicle. If the vehicle isdetected almost simultaneously by two mobile devices of two users, thetwo users are very likely to be in the same vehicle. The first andsecond filtered signal streams may additionally include velocity datafrom GPS sensors respectively. If the velocity data indicates the twousers are moving, the activity identifier 208 can estimate the two usersare travelling in the same vehicle. In another example, assume the thirddevice is a device at home with a Bluetooth sensor (e.g., aBluetooth-enabled personal computer). If the device at home isrespectively detected by two mobile devices of two users within apredetermined time window (e.g., within 10 seconds), the activityidentifier 208 can estimate that the two users are at home. In someexamples, the activity identifier 208 estimates two users as beingtogether if the location data from GPS sensors indicates the two users'geo-locations are the same.

In some embodiments, the filter engine 206 filters the signal streams toadditionally include received signal strength indicator (RSSI) data forincreased granularity. In some embodiments, the filter engine 206 maypoll for specific known devices by filtering available devices based ona social graph of a user and/or the user's location. For example, thefilter engine 206 identifies a group of devices used by the user'sfriends. In another example, the filter engine 206 identifies a group ofdevices at the same location as the user. In yet another example, thefilter engine 206 identifies a group of devices that are used by theuser's friends and at the same location as the user.

In some embodiments, the filter engine 206 provides the filtered signalstream to applications stored on the user device 115. For example, thestep data from the filtered stream is input to a fitness trackingapplication. In other embodiments, the filter engine 206 stores thefiltered signal stream in the storage device 243. In some otherembodiments, the filter engine 206 sends the filtered signal stream tothe activity identifier 208.

The activity identifier 208 can be software including routines foridentifying activities. In some embodiments, the activity identifier 208can be a set of instructions executable by the processor 235 to providethe functionality described below for identifying activities. In someembodiments, the activity identifier 208 can be stored in the memory 237of the computing device 200 and can be accessible and executable by theprocessor 235. The activity identifier 208 may be adapted forcooperation and communication with the processor 235 and othercomponents of the computing device 200 via signal line 236.

Example activities include, but are not limited to, physical activities(e.g., running, walking, sleeping, driving, talking to someone, biking,talking to a group, hiking, etc.), activities on social networks (e.g.,playing online games on a social network, publishing posts and/orcomments, acknowledging posts, sharing posts, etc.) and activities onuser devices 115 (e.g., opening an application, listening to a playlist,calling a contact, writing emails, viewing photos, watching videos,etc.). Other example activities are possible.

In some embodiments, the activity identifier 208 receives a filteredsignal stream from the filter engine 206, and identifies one or moreactivities from the filtered signal stream. For example, assume thefiltered signal stream includes step data from a pedometer. The activityidentifier 208 identifies that the user is walking if the frequency ofsteps conforms to the user's walking pace. However, if the frequency ofsteps conforms to the user's running pace, the activity identifier 208identifies that the user is running In another example, the filteredsignal stream includes (1) acceleration data indicating zeroacceleration from an accelerometer, (2) timestamp data indicating thetime is midnight from a GPS sensor, (3) brightness data indicatinglights are off from a light detector, (4) power usage indicating thatthe user device 115 is connected to a charger and (5) application usageindicating that the applications are not being used. The activityidentifier 208 identifies that the user activity is sleeping based onthe filtered signal stream.

In some embodiments, the activity identifier 208 determines useractivities based on data received from multiple virtual detectors 202and/or hardware sensors 252. For example, the filtered signal streamincludes data indicating (1) a game application is running on the userdevice 115 and (2) the user is swiping fingers on the touch screen ofthe user device 115. The activity identifier 208 identifies that theuser is playing a game on the user device 115. In another example, thefiltered signal stream includes (1) data describing steps from apedometer, (2) data describing that a music application is running onthe user device 115 from the virtual detector 202, and (3) datadescribing a friend's mobile device is detected in proximity to the userdevice 115 from a Bluetooth sensor of the user device 115. The activityidentifier 208 identifies that the user is listening to music andjogging with the friend based on the usage of the music application, thefrequency of steps and presence of the friend's mobile device inproximity to the user device 115. In yet another example, the filteredsignal stream includes (1) location data describing the user iscurrently in a coffee shop from a GPS sensor of the user device 115 and(2) data describing a friend's mobile device is detected in proximity tothe user device 115 from a Bluetooth sensor of the user device 115. Theactivity identifier 208 identifies that the user is meeting with thefriend at the coffee shop.

In some embodiments, the activity identifier 208 retrieves datadescribing a user profile from the social network server 101 withpermission from the user. The user profile includes one or more of theuser's age, gender, education background, working experience, interestsand other demographic information. The activity identifier 208identifies one or more activities associated with the user from thefiltered signal stream based on the user profile. For example, for aparticular frequency of steps determined based on the step data from apedometer, the activity identifier 208 may determine that the user isrunning if the user is a senior over 60 years old. However, the activityidentifier 208 may determine that the user is walking at a fast pace ifthe user is a young athlete. In another example, if the user iscategorized as a marathon running, the activity identifier 208 is morelikely to identify the user activity as running than other activitiessuch as biking, swimming, etc.

In some embodiments, the activity identifier 208 identifies a socialaspect, an attention aspect and/or a mobility aspect for each activitybased on the filtered signal stream. A social aspect indicates who iswith the user during the activity. For example, a social aspect of arunning activity indicates that a friend runs together with the user. Inanother example, a social aspect of a meeting indicates whether the userattends a business meeting or meets with friends. An attention aspectindicates what the user focuses on. For example, an attention aspect ofa gaming activity indicates the user focuses his or her attention on thegame application. A mobility aspect indicates a state of the user. Forexample, the mobility aspect indicates the user is sitting or movingduring the activity. In some embodiments, the mobility aspect describesthe user's geo-location. For example, the mobility aspect indicates theuser is driving on a highway.

In some embodiments, the filtered signal stream includes change inactions, and the activity identifier 208 identifies a beginning and/oran ending of an activity from the filtered signal stream. For example,at a first timestamp, the activity identifier 208 identifies a beginningof a running activity if the filtered signal stream includes dataindicating that the frequency of the user's steps increases from awalking pace to a running pace. At a second timestamp, the activityidentifier 208 identifies an ending of the running activity if thefiltered signal stream includes data indicating the frequency of theuser's steps decreases from a running pace to a walking pace. In anotherexample, at a first timestamp, the activity identifier 208 identifies abeginning of a dining activity if the filtered signal stream includes(1) location data indicating the user arrives at a restaurant and (2)data indicating presence of a friend's mobile device in proximity to theuser's mobile device. At a second timestamp, the activity identifier 208identifies an ending of the dining activity if the filtered signalstream includes location data indicating the user leaves the restaurant.

The aggregator 210 can be software including routines for aggregatingactivities associated with a user. In some embodiments, the aggregator210 can be a set of instructions executable by the processor 235 toprovide the functionality described below for aggregating activitiesassociated with a user. In some embodiments, the aggregator 210 can bestored in the memory 237 of the computing device 200 and can beaccessible and executable by the processor 235. The aggregator 210 maybe adapted for cooperation and communication with the processor 235 andother components of the computing device 200 via signal line 238.

In some embodiments, the aggregator 210 aggregates one or moreactivities associated with a user to define an event related to theuser. An event can be data describing a story of a user. In someembodiments, an event includes a single activity performed during aparticular time period. For example, an exercise event describes thatthe user ran in a park from 6:00 AM to 6:30 AM. In some embodiments, anevent includes multiple activities performed by a user during aparticular time period. For example, a Saturday social event from 3:00PM to 10:00 PM includes shopping with friends in a mall from 3:00 PM to6:00 PM, dining with the friends in a restaurant from 6:00 PM to 8:00 PMand going to a movie with the friends from 8:00 PM to 10:00 PM.

In some embodiments, an event includes multiple activities related to aparticular subject. For example, a gaming event includes playing a videogame with a friend, posting a gaming result on a social network, sharinggaming photos online and posting comments on the gaming result. In someother embodiments, an event includes one or more activities performed atthe same location. For example, a sports event includes watching asports game with friends in a stadium, taking photos of the sports game,shopping for a jersey in the stadium and encountering a colleague in thestadium, etc. Other example events are possible. In some embodiments,the aggregator 210 stores the events defined from activities of a userin the data storage 243. In other embodiments, the aggregator 210 sendsthe events to the summarizing engine 212.

The summarizing engine 212 can be software including routines forgenerating detailed summaries of events for a first user depending on atype of relationship between the first user and a second user. In someembodiments, the summarizing engine 212 can be a set of instructionsexecutable by the processor 235 to provide the functionality describedbelow for generating graphical data for generating summaries. In someembodiments, the summarizing engine 212 can be stored in the memory 237of the computing device 200 and can be accessible and executable by theprocessor 235. The summarizing engine 212 may be adapted for cooperationand communication with the processor 235 and other components of thecomputing device 200 via signal line 240.

In some embodiments, the summarizing engine 212 receives eventsincluding activities of a first user from the aggregator 210 andgenerates a summary of the first user's activities based on the events.In some embodiments, the activities include activities within a certaintime period. In some embodiments, the activities may include activitiesduring a first user's life during a period or periods of time betweeninteractions of the first user and a second user. For example, thesummarizing engine 212 receives events of the first user's physicalactivities, activities on social networks and activities on user devices115 from the aggregator 210 and generates a summary of the first user'sactivities. In some embodiments, the summarizing engine 212 generates asummary of the first user's activities during a specified time period(e.g., a day, a week or a month). For example, the summarizing engine212 generates a summary of applications used by the first user, postspublished by the first user, people meeting with the first user, photosshared by the first user, videos viewed by the first user and otherphysical activities (e.g., biking, walking etc.) performed by the firstuser during the specified time period. For example, the summarizingengine 212 may generate a summary of activities during a user'slifetime. For example, the summarizing engine 212 may summarize oridentify important moments in the lifetime of a first user during theperson of time between interactions between the first user and anotheruser.

In some embodiments, the summarizing engine 212 determines that thefirst user is within proximity to a second user, generates a summary ofthe first user's activities based on the first user's closeness with thesecond user and provides the first user with the summary. In someembodiments, the proximity is a physical proximity between the first andsecond users. The summarizing engine 212 determines that the first useris within proximity to the second user based on data received frommultiple virtual detectors 202 and/or hardware sensors 252. For example,the summarizing engine 212 determines that the first user is withinproximity to the second user based on data describing that the seconduser's mobile device is detected in proximity to the user device 115associated with the first user from a Bluetooth sensor of the userdevice 115, or based on location data describing both the first user andthe second user are currently in a coffee shop from GPS sensors of theuser devices 115 associated with the first user and the second user.

In some embodiments, the summarizing engine 212 determines the firstuser's closeness with the second user based on a connection between thefirst user and the second user. In some embodiments, the connection is asocial connection between the first user and the second user on a socialnetwork. For example, the summarizing engine 212 receives social data(e.g., profiles, relationships, a social graph, etc.) from one or moresocial networks and determine if and how users are connected. In someembodiments, the summarizing engine 212 determines the first user'scloseness with the second user based on a degree of separation of asocial connection between the first user and the second user. Forexample, the summarizing engine 212 identifies that the first and secondusers follow each other in a social network and determines a degree ofseparation of one between the first and second users. Based on thisfirst-degree separation, the summarizing engine 212 determines that thefirst user and second user are close. The lower the degree of separationis, the closer the first and second users are. For example, thesummarizing engine 212 determines that there is a first-degreefriendship connection between the first and second users since they aredirectly connected in a social network with a friendship connection. Thesummarizing engine 212 also determines that there is a second-degreefriendship connection between the first user and a third user since theyare connected in the social network via a mutual friend. The summarizingengine 212 determines that the first user is closer to the second userthan to the third user based on comparing the degrees of separation.

In other embodiments, the summarizing engine 212 determines a connectionbetween the first and second users based on other sources. For example,besides the social network connection on a social network describedabove, the sources for the connection between the first and second userscan also include communications, such as emails, micro-blogs, blogs,forums, user contact lists, corporate employee databases, organizationalcharts, etc. As another example, the sources for connection between thefirst and second users can also be historical co-presence of the firstand second users. For example, the summarizing engine 212 can determineif users are connected by checking users' contact lists or bydetermining if users have sent or received a certain number of emails(e.g., one email, five emails, 10 emails, 50 emails, etc.) to or fromeach other in a certain period of time (e.g., in a week, in a month, ina year, etc.). In another example, the summarizing engine 212 candetermine user connections by analyzing corporate employee databases orschool alumni databases, etc. For example, the summarizing engine 212determines that users are connected if they have worked for the sameemployer or if they have studied at the same school.

In some embodiments, the summarizing engine 212 determines the firstuser's closeness with the second user based on the first and secondusers' connection from other sources. For example, if the first userfrequently exchanges emails with the second user (e.g., ten emails perweek) while seldom communicating with a third user via emails (e.g., twoemails per month), the summarizing engine 212 determines that the firstuser is closer to the second user than to the third user. If the firstuser met a fourth user once when they attended the same conference oneyear ago, the summarizing engine 212 determines that the first user iscloser to the third user than to the fourth user.

In some embodiments, the summarizing engine 212 determines that thefirst user's closeness with the second user based on a time elapsedsince a last interaction. In some embodiments, the summarizing engine212 determines which events to include in the summary based on a timeelapsed since a last interaction. The summarizing engine 212 receivesactivity data from the activity identifier 208 and determines a timeelapsed since a last interaction between the first and second usersbased on the activity data. For example, the time elapsed may be timeelapsed since the last face-to-face meeting between the first and secondusers, or since the last communication between the first and secondusers, or some other last interaction between the first and secondusers. The summarizing engine 212 increases the closeness between thefirst and second users when the determined time is reduced. For example,the summarizing engine 212 determines that the first user and the seconduser are close based on an activity that the first user commented on apost sent by the second user 15 minutes ago. The summarizing engine 212determines that the first user is distant to a third user since the lastinteraction between them was that the third user replied an email fromthe first user two years ago.

In some embodiments, the summarizing engine 212 determines that thefirst user's closeness with the second user based on at least one of aconnection between the first user and the second user and a time elapsedsince a last interaction. For example, the summarizing engine 212determines that the first and second users are close since they aredirectly connected with a friendship connection on a social network(e.g., a degree of separation of one). However, if the summarizingengine 212 also determines that the last interaction between the firstand second users (e.g., an email) occurred one year ago, the summarizingengine 212 decreases the closeness between the first and second users.

The summarizing engine 212 determines relationships between the firstand second users based on the closeness. In some embodiments, thesummarizing engine 212 classifies the first user's relationship with thesecond user as being a first type of relationship, a second type ofrelationship or a third type of relationship. If the first user knowsthe second user very well (e.g., close friends and/or interact with eachother frequently), the summarizing engine 212 classifies therelationship between the first and second users as being a first type ofrelationship. If the first user does not know the second user very well(e.g., acquaintances and/or interact with each other infrequently), thesummarizing engine 212 classifies the relationship between the first andsecond users as being a second type of relationship. If the first userhas not met the second user in person (e.g., strangers with informationshared in common), the summarizing engine 212 classifies therelationship between the first and second users as being a third type ofrelationship.

In some embodiments, the summarizing engine 212 determines relationshipsbetween the first and second users based on various contexts. Forexample, in some embodiments, the summarizing engine 212 classifies thefirst user's relationship with the second user based on the context ofthe relationship. For example, the first and second users may have apersonal relationship. As another example, the first and second usersmay have a professional relationship.

In some embodiments, the summarizing engine 212 determines relationshipsbetween the first and second users based on historical context. Forexample, if the first user and the second user were on the same sportsteam, the relationship may be categorized based on that. As anotherexample, if the first user and the second user were in the same club,the relationship may be categorized based on that.

In some embodiments, the summarizing engine 212 assigns thresholddegrees of separation and determines whether the first user knows thesecond user well based on the threshold degrees of separation. Forexample, the summarizing engine 212 assigns a first threshold degree ofseparation as two and assigns a second threshold degree of separation asseven. If the degree of separation of a connection between the first andsecond users is less than two, the summarizing engine 212 determinesthat the first user knows the second user very well and they have afirst type of relationship. If the degree of separation of a connectionbetween the first and second users is between two and seven, thesummarizing engine 212 determines that the first user generally knowsthe second user (e.g., they are on edge of a social graph) and they havea second type of relationship. If the degree of separation of aconnection between the first and second users is greater than seven, thesummarizing engine 212 considers that the first and second users arestrangers and determines that they have a third type of relationship.

In other embodiments, the summarizing engine 212 uses other factors(e.g., interaction frequency, a time elapsed since a last interaction,etc.) to determine whether the first user knows the second user verywell. For example, if the summarizing engine 212 determines that thefirst and second users interact with each other on a social networkabout twice per month or determines that the last interaction betweenthe first and second users was 20 days ago, the summarizing engine 212determines that the first user does not know the second user very welland classifies the relationship between the first and second users asbeing a second type of relationship.

Responsive to having the first type of relationship, the summarizingengine 212 generates a first summary for the first user that includes anotification that the second user is nearby, a last interaction with thesecond user and recent interactions with the second user. In particular,the first summary includes any action items that the first user owes thesecond user. For example, the summarizing engine 212 includes anotification “dinner with your friend” and restaurant information in thefirst summary to remind the first user that he/she needs to make dinnerplans with the second user. In another example, the summarizing engine212 attaches a starred email in the first summary to remind the firstuser that he/she needs to give a class notes mentioned in the starredemail to the second user. The first summary will be described in detailbelow with reference to FIG. 3A.

Responsive to having the second type of relationship, the summarizingengine 212 generates a second summary for the first user that includesthe notification that the second user is nearby, the last interactionwith the second user and events that the first user and the second usershare in common. Examples of social events that the first user and thesecond user share in common may be photos of the two users together,events that both users attended, and stories related to both users.Other examples of events that the first user and the second user sharein common may be events that happened specifically in this locationwhere the two users are currently located. In some embodiments, thesecond summary includes important events that occurred to the first userthat the second user might be interested in hearing about. For example,after the first and second user last met in an information technology(IT) conference, the first user started up an IT company. Thesummarizing engine 212 includes the information of the first user's ITcompany in the second summary. In other embodiments, the second summaryalso includes a name of a mutual connection and an event that the mutualconnection attended. For example, in the above example, the summarizingengine 212 also includes the IT conference that both the first user andthe second user attended last year in the second summary to remind thefirst user of the second user. In another example, the summarizingengine 212 includes a name of a university from which the first andsecond users graduated or a mutual friend's name in the second summary.In yet another example, the summarizing engine 212 includes a commonhobby or common acquaintances between the first a second users. In yetother embodiments, the second summary includes a recent post on a socialnetwork that was created by the second user. For example, the secondsummary includes a picture of a restaurant near the Golden Gate Bridgetaken by the second user on the first day the second user arrived in SanFrancisco where the first user lives. From this picture, the first usercan get a talking point for conversing with the second user by receivinginformation about which kind of restaurants the second user likes andwhich kind of places the second user wants to visit. As a result, theconnection between the first and second users may be enhanced. Thesecond summary will be described in detail below with reference to FIGS.3B and 3C.

Responsive to having the third type of relationship, the summarizingengine 212 generates a third summary for the first user that includesthe notification that the second user is nearby and events that thefirst user and the second user share in common. For example, thesummarizing engine 212 includes biographical information that the firstuser and the second user share in common such as having worked in a samecompany, joining a New Year's celebration every year at Times Square inNew York City. The third summary will be described in detail below withreference to FIG. 3D.

Once the summarizing engine 212 determines that the first user is withinproximity to the second user, the summarizing engine 212 generates asummary based on the type of relationship between the first and secondusers and provides the first user with the summary. The summary includesa notification that the second user is nearby, a last time the firstuser interacted with the second user and information about at least oneof the first user and the second user. In some embodiments, thesummarizing engine 212 determines that the first user is withinproximity to the second user, for example, when the first and secondusers are face-to-face or close enough in proximity that they are in aconversation with each other. In some embodiments, the notification mayinclude that another user is a certain distance away. In otherembodiments, the notification could prompt the two users to coordinateonline or start heading toward one another in order to meet.

The user interface engine 214 can be software including routines forgenerating graphical or audio data for providing user interfaces tousers. In some embodiments, the user interface engine 214 can be a setof instructions executable by the processor 235 to provide thefunctionality described below for generating graphical data forproviding user interfaces to users or providing audio data to users. Insome embodiments, the user interface engine 214 can be stored in thememory 237 of the computing device 200 and can be accessible andexecutable by the processor 235. The user interface engine 214 may beadapted for cooperation and communication with the processor 235 andother components of the computing device 200 via signal line 242.

In some embodiments, the user interface engine 214 generates graphicaldata for providing a user interface that depicts a summary of a user'sactivities. The user interface engine 214 sends the graphical data to auser device 115, causing the user device 115 to present the userinterface to the user. In some embodiments, the user interface engine214 may help trigger launching of a relevant application and relevantcontent with the applications (for example, an email application mighthave the right email open or the result of a query for related emailsbetween the first and second users). Example user interfaces are shownin FIGS. 3A-3D. In other embodiments, the user interface engine 214generates graphical data for providing a user interface that depicts anevent associated with one or more users. A user may modify or update theevent notification, add more peers to the event, share the event, add adetailed description of photos, make comments on the event, add orupdate a title for the event, or perform other actions on the eventusing the user interface. The user interface engine 214 may generategraphical data for providing other user interfaces to users.

The privacy engine 216 can be software including routines fordetermining what information to provide to a user based on privacysettings. In some embodiments, the privacy engine 216 can be a set ofinstructions executable by the processor 235 to provide thefunctionality described below for determining what information toprovide to a user based on privacy settings, the privacy engine 216 canbe stored in the memory 237 of the computing device 200 and can beaccessible and executable by the processor 235. The privacy engine 216may be adapted for cooperation and communication with the processor 235and other components of the computing device 200 via signal line 217.

In some embodiments, the privacy engine 216 determines privacy settingsfrom a user profile associated with a user. For example, John manuallyselects privacy settings such as preferring to share personal photoswith a group in the social network that he created called “closefriends.” The privacy engine 216 communicates with the summarizingengine 212 to determine what information to provide to a user based onprivacy settings. For example, in the above example, when John is nearbyhis best friend Linda and his acquaintance Sara, the summary engine 212generates a first summary for Linda that includes a personal photo ofJohn and generates a second summary for Sara without including thepersonal photo. As another example, the summary engine 212 may alsogenerate a third summary of the common items for all three users of allof the three users together (for example, email threads between allthree users, documents that all three users collaborated on, or sharedphotos among the three users, etc.).

FIG. 3A is an example graphic representation of a user interface 300 fordisplaying a summary where the first user and the second user are closefriends and/or interact with each other frequently. The user interface300 displays a summary generated for Lance. The user interface 300includes a notification 301 notifying Lance that Bob Smith is nearby.Bob is a close friend of Lance. Four days ago they met with each otherat a coffee shop. The user interface 300 includes this last interactionat 302. Besides this interaction, the user interface 300 also includesother interactions between Bob and Lance, for example, the indication303 shows that Bob and Lance had a two-minute call two weeks ago and theindication 304 shows that Bob and Lance exchanged an email about aproject three weeks ago. Using the indication 304, Lance may be remindedthat he owns some documents related to the project to Bob.

FIGS. 3B and 3C are example graphic representations of user interfaces320 and 340 for displaying a summary where the first user and the seconduser are friends and/or interact with each other infrequently. The userinterfaces 320 and 340 display a first portion and a second portion of asummary generated for Lance. In FIG. 3B, the user interface 320 includesa notification 321 notifying Lance that Sara Doe is nearby. Sara is afriend of Lance. Last time they met with each other was at Oren'sBarbeque one year ago. The user interface 320 includes this lastinteraction at 322. The user interface 320 also includes events thatSara and Lance share in common, for example, the indication 323 showsthat Sara and Lance both attended Noname University, the indication 324shows that Sara and Lance are both friends of John Doe, the indication325 shows that Sara and Lance both live in San Francisco, Calif. and theindication 326 shows that Sara and Lance used to live in Cambridge,Mass.

Referring now to FIG. 3C, the user interface 340 displays a secondportion of the summary generated for Lance. The user interface 340includes a recent post 341 on a social network that was created by Sara.The picture 341 taken by Sara shows how Sara was scared by a bear whenthe bear was too close to her. The user interface 340 also includesimportant events that occurred to Lance that Sara might be interested inhearing about, e.g., Lance's life since last communicating with Sara342. For example, the picture 343 shows that Lance had a baby, thepicture 344 shows that Lance started a company and the picture 345 showsthat Lance had lunch with their mutual friend Jenny.

FIG. 3D is an example graphic representation of a user interface 360 fordisplaying a summary where the first user and the second user arestrangers with information shared in common. The user interface 360displays a summary generated for Lance. The user interface 360 includesa notification 361 notifying Lance that Mike Jones is nearby. Lance doesnot really know Mike but shares common information with Mike. Forexample, the indications 362, 363 and 364 show that both Lance and Mikeare friends of Alice Doe, used to work in X company and currently livein San Diego, Calif. By providing the common information shared betweenLance and Mike, they have some topics to discuss and may eventually knoweach other well.

FIG. 4 is a flow diagram of an example of a method for generating asummary for a first user. In some embodiments, the summary application103 comprises a processing unit 204, a filter engine 206, an activityidentifier 208 and a summarizing engine 212. The processing unit 204receives 402 a signal stream from at least one of a hardware sensor 252and a virtual detector 202. The filter engine 206 filters 404 the signalstream and outputs a filtered signal stream including data for definingone or more human-understandable actions. The activity identifier 208identifies 406 one or more activities associated with a first user fromthe filtered signal stream. The summarizing engine 212 generates 408 asummary of the first user's activities. The summarizing engine 212determines 410 that the first user is within proximity to a second user.The summarizing engine 212 determines 412 the first user's closenesswith the second user based on at least one of a connection between thefirst user and the second user and a time elapsed since a lastinteraction. The summarizing engine 212 provides 414 the first user witha notification that the second user is nearby, a last time the firstuser interacted with the second user and information about at least oneof the first user and the second user.

FIGS. 5A and 5B are flow diagrams of another example of a method forgenerating a summary for a first user depending on the type ofrelationship between the users. In some embodiments, the summaryapplication 103 comprises a processing unit 204, a filter engine 206, anactivity identifier 208, an aggregator 210 and a summarizing engine 212.In FIG. 5A, the processing unit 204 receives 502 a signal stream from atleast one of a hardware sensor 252 and a virtual detector 202. Thesignal stream includes at least one of hardware raw data generated bythe hardware sensor 252 and virtual raw data generated by the virtualdetector 202. The filter engine 206 filters 504 the signal stream andoutputs a filtered signal stream including data for defining one or morehuman-understandable actions. For example, the filter engine 206 outputsa filtered signal stream that combines one or more of the following dataincluding: (1) location and velocity data from a GPS sensor, and (2)detection data indicating presence of an automobile (e.g., Bluetoothenabled) and a mobile in close proximity, etc. to indicate travellingtogether with another user.

The activity identifier 208 identifies 506 one or more activitiesassociated with a first user from the filtered signal stream. Forexample, for a particular frequency of steps determined based on thestep data from a pedometer, the activity identifier 208 may determinethat the user is running if the user is a senior over 60 years old.However, the activity identifier 208 may determine that the user iswalking at a fast pace if the user is a young athlete. In anotherexample, if the user is categorized as a marathon running, the activityidentifier 208 is more likely to identify the user activity as runningthan other activities such as biking, swimming, etc.

The summarizing engine 212 generates 508 a summary of the first user'sactivities. In some embodiments, the summarizing engine 212 receivesevents including activities of a first user from the aggregator 210 andgenerates a summary of the first user's activities based on the events.The summarizing engine 212 determines 510 that the first user is withinproximity to a second user. For example, the summarizing engine 212determines that the first user is within proximity to the second userbased on data describing that the second user's mobile device isdetected in proximity to the user device 115 associated with the firstuser from a Bluetooth sensor of the user device 115.

The summarizing engine 212 determines 512 a degree of separation betweenthe first user and the second user in a social network. For example, thesummarizing engine 212 identifies that the first and second users followeach other in a social network and determines a degree of separation ofone between the first and second users. In this example, based on thisfirst-degree separation, the summarizing engine 212 determines that thefirst user and second user are close.

The summarizing engine 212 determines 514 a time elapsed since a lastinteraction between the first user and the second user. For example, thesummarizing engine 212 determines that the first user and the seconduser are close based on an activity that the first user commented on apost sent by the second user 15 minutes ago. The summarizing engine 212determines that the first user is distant to a third user since the lastinteraction between them was that the third user replied an email fromthe first user two years ago. In some embodiments, the summarizingengine 212 determines 514 a time elapsed since a last interactionbetween the first user and the second user to determine relevant eventsto include.

Referring now to FIG. 5B, the summarizing engine 212 classifies 516 thefirst user's relationship with the second user as being a first type ofrelationship, a second type of relationship or a third type ofrelationship. If the first user knows the second user very well (e.g.,close friends and/or interact with each other frequently), thesummarizing engine 212 classifies the relationship between the first andsecond users as being a first type of relationship. If the first userdoes not know the second user very well (e.g., acquaintances and/orinteract with each other infrequently), the summarizing engine 212classifies the relationship between the first and second users as beinga second type of relationship. If the first user has not met the seconduser in person (e.g., strangers with information shared in common), thesummarizing engine 212 classifies the relationship between the first andsecond users as being a third type of relationship.

Responsive to having the first type of relationship, the summarizingengine 212 generates 518 a first summary for the first user thatincludes a notification that the second user is nearby, a lastinteraction with the second user and recent interactions with the seconduser. For example, the summarizing engine 212 includes a notification“dinner with your friend” and restaurant information in the firstsummary to remind the first user that he/she owns a dinner to the seconduser.

Responsive to having the second type of relationship, the summarizingengine 212 generates 520 a second summary for the first user thatincludes the notification that the second user is nearby, the lastinteraction with the second user and events that the first user and thesecond user share in common. In some embodiments, the second summaryincludes important events that occurred to the first user that thesecond user might be interested in hearing about. In other embodiments,the second summary also includes a name of a mutual connection and anevent that the mutual connection attended. In yet other embodiments, thesecond summary includes a recent post on a social network that wascreated by the second user.

Responsive to having the third type of relationship, the summarizingengine 212 generates 522 a third summary for the first user thatincludes the notification that the second user is nearby and events thatthe first user and the second user share in common. For example, thesummarizing engine 212 includes biographical information that the firstuser and the second user share in common such as having worked in a samecompany, joining New Year's celebration every year at Times Square inNew York City.

In the above description, for purposes of explanation, numerous specificdetails are set forth in order to provide a thorough understanding ofthe specification. It will be apparent, however, to one skilled in theart that the invention can be practiced without these specific details.In other instances, structures and devices are shown in block diagramform in order to avoid obscuring the description. For example, thepresent embodiment is described in one embodiment below primarily withreference to user interfaces and particular hardware. However, thepresent embodiment applies to any type of computing device that canreceive data and commands, and any peripheral devices providingservices.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the description. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment.

Some portions of the detailed descriptions that follow are presented interms of algorithms and symbolic representations of operations on databits within a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present embodiment of the specification also relates to an apparatusfor performing the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may comprise ageneral-purpose computer selectively activated or reconfigured by acomputer program stored in the computer. Such a computer program may bestored in a computer readable storage medium, such as, but is notlimited to, any type of disk including optical disks, CD-ROMs, andmagnetic disks, read-only memories (ROMs), random access memories(RAMs), EPROMs, EEPROMs, magnetic or optical cards, flash memoriesincluding USB keys with non-volatile memory or any type of mediasuitable for storing electronic instructions, each coupled to a computersystem bus.

The specification can take the form of an entirely hardware embodiment,an entirely software embodiment or an embodiment containing bothhardware and software elements. In a preferred embodiment, thespecification is implemented in software, which includes but is notlimited to firmware, resident software, microcode, etc.

Furthermore, the description can take the form of a computer programproduct accessible from a computer-usable or computer-readable mediumproviding program code for use by or in connection with a computer orany instruction execution system. For the purposes of this description,a computer-usable or computer readable medium can be any apparatus thatcan contain, store, communicate, propagate, or transport the program foruse by or in connection with the instruction execution system,apparatus, or device.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers.

Network adapters may also be coupled to the system to enable the dataprocessing system to become coupled to other data processing systems orremote printers or storage devices through intervening private or publicnetworks. Modems, cable modem and Ethernet cards are just a few of thecurrently available types of network adapters.

Finally, the algorithms and displays presented herein are not inherentlyrelated to any particular computer or other apparatus. Variousgeneral-purpose systems may be used with programs in accordance with theteachings herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps. The requiredstructure for a variety of these systems will appear from thedescription below. In addition, the specification is not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the specification as described herein.

The foregoing description of the embodiments of the specification hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the specification to theprecise form disclosed. Many modifications and variations are possiblein light of the above teaching. It is intended that the scope of thedisclosure be limited not by this detailed description, but rather bythe claims of this application. As will be understood by those familiarwith the art, the specification may be embodied in other specific formswithout departing from the spirit or essential characteristics thereof.Likewise, the particular naming and division of the modules, routines,features, attributes, methodologies and other aspects are not mandatoryor significant, and the mechanisms that implement the specification orits features may have different names, divisions and/or formats.Furthermore, as will be apparent to one of ordinary skill in therelevant art, the modules, routines, features, attributes, methodologiesand other aspects of the disclosure can be implemented as software,hardware, firmware or any combination of the three. Also, wherever acomponent, an example of which is a module, of the specification isimplemented as software, the component can be implemented as astandalone program, as part of a larger program, as a plurality ofseparate programs, as a statically or dynamically linked library, as akernel loadable module, as a device driver, and/or in every and anyother way known now or in the future to those of ordinary skill in theart of computer programming. Additionally, the disclosure is in no waylimited to implementation in any specific programming language, or forany specific operating system or environment. Accordingly, thedisclosure is intended to be illustrative, but not limiting, of thescope of the specification, which is set forth in the following claims.

What is claimed is:
 1. A computer-implemented method comprising:receiving, with one or more processors, a signal stream from at leastone of a hardware sensor and a virtual detector; filtering, with the oneor more processors, the signal stream and outputting filtered signalstream including data defining human-understandable actions; identifyingactivities associated with a first user from the filtered signal stream;generating a summary of the first user's activities; determining thatthe first user is within proximity to a second user; determining adegree of separation between the first user and the second user in asocial network; determining a time elapsed since a last interactionbetween the first user and the second user; classifying the first user'srelationship with the second user as being a first type of relationship,a second type of relationship or a third time of relationship;responsive to having the first type of relationship, generating a firstsummary for the first user that includes a notification that the seconduser is nearby, a last interaction with the second user and recentinteractions with the second user; responsive to having the second typeof relationship, generating a second summary for the first user thatincludes the notification that the second user is nearby, the lastinteraction with the second user and events that the first user and thesecond user share in common; and responsive to having the third type ofrelationship, generating a third summary for the first user thatincludes the notification that the second user is nearby and events thatthe first user and the second user share in common.
 2. Acomputer-implemented method comprising: receiving, with one or moreprocessors, a signal stream from at least one of a hardware sensor and avirtual detector; filtering, with the one or more processors, the signalstream and outputting filtered signal stream including data defininghuman-understandable actions; identifying activities associated with afirst user from the filtered signal stream; generating a summary of thefirst user's activities; determining that the first user is withinproximity to a second user; determining the first user's closeness withthe second user based on at least one of a connection between the firstuser and the second user and a time elapsed since a last interaction;and provide the first user with a notification that the second user isnearby, a last time that the first user interacted with the second userand information about at least one of the first user and the seconduser.
 3. The method of claim 2, wherein the summary includes actionitems that the first user owes the second user.
 4. The method of claim2, wherein the summary includes important events that occurred to thefirst user that the second user might be interested in hearing about. 5.The method of claim 2, wherein the summary includes a name of a mutualconnection and an event that the mutual connection attended.
 6. Themethod of claim 2, wherein the summary includes a recent post on asocial network that was created by the second user.
 7. The method ofclaim 2, wherein the first user and the second user have not met and theinformation about the last time the first user interacted with thesecond user is a shared event that they attended.
 8. The method of claim2, further comprising determining closeness between the first user andthe second user based on at least one of the degree of separation andthe time elapsed since the last interaction, and wherein classifying thefirst user's relationship with the second user is based on thecloseness.
 9. The method of claim 1, further comprising determining whatinformation to provide in the summary to the first user about the seconduser based on privacy settings.
 10. The method of claim 1, whereindetermining that the first user is within proximity to the second useris based at least in part on data received from at least one of thehardware sensor and the virtual detector.
 11. A system comprising: oneor more processors; a processing unit stored on a memory and executableby the one or more processors, the processing unit configured to receivea signal stream from at least one of a hardware sensor and a virtualdetector; a filter engine coupled to the processing unit and configuredto filter the signal stream and outputting filtered signal streamincluding data defining human-understandable actions; an activityidentifier coupled to the filter engine and configured to identifyactivities associated with a first user from the filtered signal stream;and a summary engine coupled to the activity identifier and configuredto generate a summary of the first user's activities, determine that thefirst user is within proximity to a second user, determine the firstuser's closeness with the second user based on at least one of aconnection between the first user and the second user and a time elapsedsince a last interaction.
 12. The system of claim 11, wherein thesummary includes action items that the first user owes the second user.13. The system of claim 11, wherein the summary includes importantevents that occurred to the first user that the second user might beinterested in hearing about.
 14. The system of claim 11, wherein thesummary includes a name of a mutual connection and an event that themutual connection attended.
 15. The system of claim 11, wherein thesummary includes a recent post on a social network that was created bythe second user.
 16. The system of claim 11, wherein the first user andthe second user have not met and the information about the last time thefirst user interacted with the second user is a shared event that theyattended.
 17. The system of claim 11, wherein determining closenessbetween the first user and the second user is based on at least one ofthe degree of separation and the time elapsed since the lastinteraction, and wherein classifying the first user's relationship withthe second user is based on the closeness.
 18. The system of claim 11,further comprising a privacy engine coupled to the summarizing engine,the privacy engine determining what information to provide in thesummary to the first user about the second user based on privacysettings.
 19. The system of claim 11, wherein determining that the firstuser is within proximity to the second user is based at least in part ondata received from at least one of the hardware sensor and the virtualdetector.